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Abstract 

We study the following coupled Schrodinger equations which have appeared as 
several models from mathematical physics: 

-Ami + Aiui = /Uilff + j3u±U2, x S Q, 
-Au 2 + A 2 «2 = M2""2 + Pu\u 2 , X&Q, 
Ui = U2 = on d£l. 

Here fl is a smooth bounded domain in M. N (N = 2,3) or il — M. N , Ai, A2, 
fii, \i% are all positive constants and the coupling constant /3 < 0. We show 
that this system has infinitely many sign-changing solutions. We also obtain 
infinitely many semi-nodal solutions in the following sense: one component 
changes sign and the other one is positive. The crucial idea of our proof, 
which has never been used for this system before, is turning to study a new 
problem with two constraints. Finally, when f2 is a bounded domain, we 
show that this system has a least energy sign-changing solution, both two 
components of which have exactly two nodal domains, and we also study the 
asymptotic behavior of solutions as j3 — > —00 and phase separation is expected. 
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1 Introduction 



In this paper we study solitary wave solutions of the coupled Gross-Pitaevskii 
equations (cf. [8]): 

= A$ 1 + Ail |$ 1 | 2 $ 1 + /3|$ 2 | 2 $ 1 , xe!l,J>0 ! 

< -if $ 2 = A$ 2 + A*2|*2| a *2 + j8|*i| 2 *a, xeO, t>0, 
' i^^tJeC, j — 1,2, 
^j(x,t) = 0, x € on, t > 0, j = 1,2, 

where = R Ar (iV = 2,3) or fi C 1^ is a smooth bounded domain, i is the 
imaginary unit, \i\,[i2 > and /? 7^ is a coupling constant. System 
arises in mathematical models from several physical phenomena, especially in 
nonlinear optics. Physically, the solution denotes the j th component of the 
beam in Kerr-like photorefractive media (cf. [1]). The positive constant fij is 
for self-focusing in the j th component of the beam, and the coupling constant 
P is the interaction between the two components of the beam. Problem (|l.l|l 
also arises in the Hartree-Fock theory for a double condensate, i.e., a binary 
mixture of Bose-Einstein condensates in two different hyperfine states |1) and 
1 2) (cf. [2]). Physically, $j are the corresponding condensate amplitudes, jij 
and /3 are the intraspecies and interspecies scattering lengths. Precisely, the 
sign of fij represents the self-interactions of the single state \j). If fij > as 
considered here, it is called the focusing case, in opposition to the defocusing 
case where /ij < 0. Besides, the sign of f3 determines whether the interactions 
of states |1) and |2) are repulsive or attractive, i.e., the interaction is attractive 
if > 0, and the interaction is repulsive if /3 < 0, where the two states are in 
strong competition. 

To obtain solitary wave solutions of the system we set $j(x,t) = 

e tXjt Uj(x) for j — 1,2. Then system (JTTTJ) is reduced to the following elliptic 
system 

— Au± + X1U1 = fiiu\ + j3u\u\, x € O, 
< -Ait 2 + X 2 u 2 = H2U2 + P u i u 2, (1-2) 
ui = U2 = on dCl. 

Definition 1.1. We call a solution (111,112) nontrivial if Uj ^ for j = 1, 2, a 

solution (111,112) semi-trivial if (111,112) is type o/(ui,0) or (O,^)- A solution 
(111,112) is called positive if Uj > in Q for j = 1,2, a solution (111,112) sign- 
changing if both i*i and 112 change sign, a solution (1*1,1*2) semi-nodal if one 
component is positive and the other one changes sign. 

In the last decades, system (|1.2p has received great interest from many math- 
ematicians. When is the entire space M. N , the existence of least energy and 
other finite energy solutions of (TT5J was studied in [21 El [H [TO1 HH HH1 [TOl HOI 
[211 [26J [30] and references therein. In particular, when (3 > is sufficiently large, 
multiple radially symmetric sign-changing solutions of (|1.2[) were obtained in 
[21] , where the radial symmetry of M. N plays a crucial role in their proof. Under 
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assumptions Aj > 0, jUj > and /3 < 0, Liu and Wang [TO] proved that system 
(|1.2p has infinitely many nontrivial solutions. In fact, they studied a general 
m-coupled system (m > 2). Remark that whether solutions obtained in [TO] are 
positive or sign-changing is not known. 

When C M. N (N = 2, 3) is a smooth bounded domain, there are also many 
papers studying (|1.2p . Lin and Wei [T7] proved that a least energy solution of 
(|1.2I) exists within the range /3 € (— oo,/3 ), where < /?o < In case 

where Ai = A2 > 0, /ii = ^2 > and [3 < —fii, Dancer, Wei and Weth [13] 
proved the existence of infinitely many positive solutions of (|1.2[) , while the same 
result was proved for the case Ai = A2 < by Noris and Ramos [23]. When 
Q is a ball, an interesting multiplicity results on positive radially symmetric 
solutions was given in [31]. Remark that, since Ai = A2 and [i\ = jji^, so system 
(|1.2p is invariant under the transformation (141,1(2) i-> (u2,u\), which plays a 
crucial role in [TOJ [Ml IS]- Later, by using a global bifurcation approach, the 
result of [3T] was reproved by [3] without requiring the symmetric condition 
Mi = M2j but in their proof the assumption Ai = A2 plays a crucial role. Under 
assumptions Aj > 0, fa > and /3 < without requiring Ai = A2 or fj,\ = fi 2 , 
Sato and Wang [27] proved that system (|1.2I) has infinitely many semi-positive 
solutions (i.e., at least one component is positive). Note that all the papers 
mentioned above deal with the subcritical case N < 3. Recently, Chen and Zou 
[9] studied the existence and properties of least energy solutions of (|1.2j) in the 
critical case N = 4. 

In a word, for N = 2,3, a natural question, which seems to be still open for 
both the entire space case and the bounded domain case, is whether hl.2\) has 
infinitely many sign- changing solutions when (3 < 0. This is expected by many 
experts but no proof has yet been obtained. Here we can give a positive answer 
to this open question. Since the results in the entire space case are slightly 
different from those in the bounded domain case, in this section we only state 
our results in the bounded domain case for the sake of brevity. The results in 
the entire space case will be given in Section 6. Our first result is as follows. 

Theorem 1.1. Let N = 2,3, C I 1 *' is a smooth bounded domain, Ai,A2, 
Mij/^ > and f3 < 0. Then hi. 2}) has infinitely many sign-changing solutions 
(tin,i) 1*^,2) such that 



Remark 1.1. Comparing with £?l Iffffl Iff If where infinitely many positive 
solutions were obtained, we do not need any symmetric assumptions Ai = A2 or 
Mi = M2- 

Remark 1.2. All the papers mentioned above and this paper deal with the fo- 
cusing case MI7M2 > 0. For the defocusing case 1^1,^2 < 0, related results can 
be seen in '8, 24, 25, 28, 291. In particular, Tavares and Terracini f28f studied 
the following general m-coupled system 



\\ u n.l\\L°°(n) + |kn,2|U°°(ft) ~> +°° as n ~> +°°- 




(1.3) 



3 



where CI is a smooth bounded domain, (3 < and fij < 0. Then \28\ Theorem 
1.1] says that for each fixed f3 < and fii, ■ ■ ■ ,fj, m < 0, there exist infinitely 
many A = ■•• , Kn,p) S R m and u = (ut, ■ ■ ■ , u m ) G Hq (CI, M. m ) such that 

(u, A) are sign-changing solutions of il.3\ ). That is, for each fixed /3 < and 
Mil ' ' ' < ; is noi fixed a priori and appears as a Lagrange multiplier 
in '281. Our result is different from \28l Theorem 1.1] in two aspects: one is 
that we deal with the focusing case [ij > 0, the other one is that and f3 

are all fixed constants in Theorem \l.ll To the best of our knowledge, our result 
for system il.2\) is new. 

Definition 1.2. A nontrivial solution (wi,it2) is called a least energy solution, 
if it has the least energy among all nontrivial solutions. A sign-changing solution 
(ui,U2) is called a least energy sign-changing solution, if it has the least energy 
among all sign- changing solutions. 

As pointed out above, for j3 € (— oo,/3o)> Lin and Wei [T7] proved that (11.21) 
has a least energy solution which turns out to be a positive solution. Since (|1.2I) 
has infinitely many sign-changing solutions for any (3 < 0, then another natural 
question is whether (11.21) has a least energy sign-changing solution, which has 
not been studied before. Here we can prove the following result. 

Theorem 1.2. Let assumptions in Theorem \l.l\ hold. Then U.2]) has a least 
energy sign-changing solution (u\,U2). Moreover, both u\ and ui have exactly 
two nodal domains. 

Theorems 11.11 and 1 1 . 21 are both concerned with sign-changing solutions. Be- 
sides positive solutions (obtained in [31[T31I3I]) and sign-changing solutions, as 
defined in Dcfinition ll.il it is natural to suspect that (|1.2[> may have semi- nodal 
solutions. Here we can prove the following result. 

Theorem 1.3. Let assumptions in Theorem ] 1.1\ hold. Then \1.2]) has infinitely 
many semi-nodal solutions {(it n .i, w n .2)}n>2 such that 

(1) u n ,i changes sign and u n ^. is positive; 

(2) ||u„,i|| L ~(n) + ||«n,2||i«>(j2) -> +oo (isii^ +oo; 

(3) u n ^\ has at most n nodal domains. In particular, U2,i has exactly two 
nodal domains, and (^2,1, ^2,2) has the least energy among all nontrivial 
solutions whose first component changes sign. 

Remark 1.3. Recently, we found that \21\ Theorem 0.1] proved that H1.2\) has 
infinitely many semi-nodal solutions for any j3 € (— ^//Tf/iJ, 0). Theorem \1.3\ 
improves \21\ Theorem 0.1] in two aspects: one is that we can obtain infinitely 
many semi-nodal solutions for (3 < —^Jlifl^; the other one is that, in \21\ no 
properties of the form (3) can be obtained by their approach. Our proofs in this 
paper are completely different from \27(. 
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Remark 1.4. Similarly, we can prove that U.2\) has infinitely many semi-nodal 
solutions {(w ra ,i) v n ,2)}n>2 such that v n ^i is positive, u„,2 changes sign and has 
at most n nodal domains. In the symmetric case where Ai = A2 and fix = fi 2 , 
(u nj i,u ny 2) obtained in Theorem and {v n .\,v n .2) may be the same solution 
in the sense of u n> \ = v„,2 and u n ^2 = However, if either Ai 7^ A2 or 

A*i 7^ l l 2, then (u n ,x,u n> 2) and (v n> i,v n> 2) are really different solutions. 

The rest of this paper proves these theorems. We give some notations here. 
Throughout this paper, we denote the norm of L p (f2) by \u\ p — (J n \u\ p dx)p , 
the norm of H^fl) by ||it|| 2 = J n (\Vu\ 2 +u 2 ) dx and positive constants (possibly 
different in different places) by C, Co, C\, ■ ■ ■ . Denote 



for convenience. Since we assume Ai, A2 > here, so || • are equivalent norms 
to || -||. Define H := ff x (0) X (0) with norm ||(ui,w 2 )||^ := IM)^ + \\u 2 \\l 2 . 

The rest of this paper is organized as follows. In Section 2 we give the proof 
of Theorem 11.11 The main idea of this proof is inspired by [28j , where a new 
notion of vector genus introduced by [28] will be used to define appropriate 
minimax values. Some arguments in our proof are borrowed from [2 8) with 
modifications. Remark that the ideas in |28j can not be used directly, and here 
we will give some new ideas. For example, to obtain nontrivial solutions of 
(|1.2|) . the crucial idea in this paper is turning to study a new problem with 
two constraints. Somewhat surprisingly, up to our knowledge, this natural idea 
has never been used for (|1.2j) in the literature. In Section 3 we will use general 
Nehari type manifolds to prove Theorem 11.21 By giving some modifications to 
arguments in Sections 2 and 3, we will prove Theorem II .31 in Section 4. 

In Section 5, we will study the limit behavior of solutions obtained here as 
j3 — > —00 by applying results in |25j directly. It turns out that components 
of the limiting profile tend to separate in different regions. This phenomena, 
called phase separation, has been well studied for _L°°-bounded positive solutions 
of (TP1) in the case N = 2, 3 by [31j [3l [25]. For other kinds of elliptic systems 
with strong competition, phase separation has also been well studied, we refer 
to [12] and references therein. The main result of Section 5 is Theorem 15.11 

Finally in Section 6, we will introduce existence results of infinitely many 
radially symmetric sign-changing and semi-nodal solutions in the entire space 
case. The main results are Theorems 16.11 and 16.21 The main ideas of the proof 
are the same as those in Sections 2-4. However, we will see that some ideas and 
arguments arc quite different from those in the bounded domain case. 



In the sequel we assume that assumptions in Theorem 11.11 hold. It is well 
known that solutions of (|1.2[) correspond to the critical points of C 2 functional 




2 Proof of Theorem 11.11 
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Ep : H — > K given by 

^(«l,«2) =i]|-"l||Ax + l\\Ml 2 -\(^Wl\i + ^\u 2 \t) + ^ J^ulul (2.1) 

Since we are only concerned with nontrivial solutions, we denote H := {(u\, 112) € 
H : Ui 7^ for i = 1, 2}, which is an open subset of H . Write u — [u\, 1*2) for 
convenience. 

Lemma 2.1. For any (ux,u 2 ) G -ff, «/ 

l/3| 2 ( I u\ul] > iiiH2\ux\t\u2\t (2.2) 

then 



sup Ep{y/tlu\, VhU2) = +00. 
ti,t 2 >0 

Proof. By (|2.2I) there exists a > such that 

u\u\ > fii\ui\i, I u\u\ > M2IM2I4, 

a In 



which implies 



Ej3(Viui,Vaiu2) =-t||wi||^ + -a;i||w 2 ||| 2 



-i(tVi|Mil4 + a^ 2 M2|"2|l)+^ai 2 |/3| / U 2 U 2 

^KIlL + |«*1I^2||a 2 -> +°° as * -> +°°- 

This completes the proof. □ 

Lemma 2.2. For any u = (1x1,1*2) € ff, «/ 

2 



i/ien system 



\\uif Xl =*iA*i|«i|4-*2|j8|/n u i u a 
||m 2 ||a 2 = i 2 ^2 |w 2 II ~ ti |/3| J n 



(2.3) 



(2.4) 



/ias a unique solution 



, / -.N _ M2|M2|4ll"llb 1 +l^lll M 2|| A2 Jg "l" 2 p. 

,^ _ Mil«illll"2||| 2 + |ff|||m||i 1 / nU M ^ ^'^ 

t2W - MiW|ui|J|u a |i-|^|«(j nU i U |)!' > U ' 
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Moreover, 

sup Ep (Vhui, ^/hu 2 ) = Ep [y/ti(u)ui, y / t 2 {u)u 2 ) 

*i,t 2 >0 v ' 

_ i^Kl!l|ui|ll 1 +2|/3|||u 1 || 2 A j| U2 || 2 A j^? M | + m Kil|| M2 ||l 2 



(2.6) 



4 M 2 \ux\i\u 2 \\-\l3nS n u{uly 
and (ti(u),t 2 (u)) is the unique maximum point of Ep{^/tiU\, \/t 2 u 2 ). 
Proof. It suffices to prove (|2.6j) . Recall that (ti(u),t 2 (u)) is the solution of 



2.4)1 . we deduce that 



h(u) Jn t 2 (u) 



t 2 (u) 



Hence for any ii, <2 > 0, 

Ep (Vhui,Vhu 2 ) =2*iIKIIai + ^2||m2||a 2 



< 



^1 ^1 A II || 2 ffa t 2 \ I, j 



<7(*i(S)ll«i|lAi+*a(«0ll"allA,) 



'/3 (Vii(w)wi, \/t 2 (u)u 2 



Therefore (|2.6p holds and (ti(u),t 2 (u)) is also the unique maximum point of 
Ep{\fhui,^/hu2) in [0,+oo) 2 . □ 

Define 

F:={H£fl : |ui| 4 >l/2, |u 2 | 4 >l/2}; 
.M^g := {it G A4* : u satisfies (pT5|) } ; 

-M^ — juSM* : /UiM2 - l^l 2 ^ ufufj > j ; 

M := {we H : K| 4 = 1, \u 2 \ 4 = 1} , M^, :=M(lM;. (2.7) 

Then = n M*g*. It is easy to check that M* , .M^, .M^,* are all open 
subsets of H and .M is closed. Define a new functional Jp : Ai* — > (0, +oo] by 

1 M2||«l|ll 1 +2|/3|||«il| 2 Al ||n 2 ||^ 2 f n ulul+^\\u 2 \\i 2 

-oo ifueX*\7W;*. 
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By the Sobolev inequality 

\\u\\ 2 ^>C\u\l VueJ#(fi), i = l,2, (2.8) 

where C is a positive constant, it is easy to check that Jg is continuous on 
M* and infjn* ^ > Ci > for some constant C\ independent of f) < 0. In 
particular, by a direct computation and recalling (|2.5I) . it is easy to check that 

J^eC 1 (A^*, (0,+oo)), 

and 

J' f} (u)(cp,0)=ti(u) / (VmiVv? + Ai«iv?) + h{u)t 2 (u)\p\ / m^, (2.9) 
Jo Jn 

j' fj (ti)(Q,ip) = t 2 (u) [ (Vu 2 V^ + \ 2 u 2 tP)+t 1 (u)t 2 {u)\p\ [ u\u 2 i> (2.10) 
Jn Jn 

hold for any u G and ip, tp G Hq(^) (Remark that ([2~^) - (|2~TU| do not hold 
for m € .M^* \ Ad/s). Note that Lemmas [231 and |2~21 yield 

Jp{u 1 ,u 2 ) = sup iJg (Vtiwx, Vhu 2 ) , V(«i,u 2 )6M. (2-H) 

*l,t 2 >0 

To obtain nontrivial solutions of (|1.2[) . we turn to study the functional Jp re- 
stricted to Mp, which is a problem with two constraints. 

Remark 2.1. To obtain nontrivial solutions of il.2)) . in many papers (see W) 
\13) \16) \17) \2b\ \31f for example ), people usually turn to study nontrivial critical 
points of Ep under the following Nehari manifold type constraint 

{(u u u 2 ) G H : E' j3 (u 1 ,u 2 )(u 1 ,0) = E' p ( Ul ,u 2 )(0,u 2 ) = 0}, 

which is actually a natural constraint for any (3 < ^J[i\[i 2 (see \2b\ Proposition 

I. 1] for example). To the best of our knowledge, our natural idea (i.e., to obtain 
nontrivial solutions of il.2)) by studying J^\m^)i has never been introduced for 

II. 2)) in the literature. 

In the following, we always let — (1,2) or (i,j) — (2,1). Recall that 

U(u) is well defined for u e M*p. For any u = (ui,u 2 ) G M*p, let w, G Hq(CI) 
be the unique solution of the following linear problem 

- Awi + \iWi + \f3\tj(u)u?Wi = mti{u)u\, Wi G Hq{Q). (2.12) 

Since \1n\4 > 1/2, so Wi =^ and 




Define 

Wi = ctiWi, where ai — -7 — 5 — > 0. (2-13) 

\n u >i 
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Then Wi is the unique solution of the following problem 

= mtr>\ 

(2.14) 



-Awi + \wi + \P\tj(u)u?Wi = aijj,iti(u)uf, w { € H%(to), 



1 J n ufwi dx = 1. 
Now we define an operator K = (K\,K2) ■ M.% — > H by 

K(u) = (K^K^u)) :=vS={w 1 ,w 2 ). (2.15) 
Define the transformations 

a : H ->• H by ai(ux,u 2 ) := (-tti,w 2 ), 02(1/1, tt 2 ) := -1/2)- (2.16) 
Then it is easy to check that 

Kfaffl) = ai{K{u)), » = 1,2. (2.17) 
Lemma 2.3. K G C X {M},H). 

Proof. It suffices to apply the Implicit Theorem to the C 1 map 
* : M* p x ff x (O) x R — > F x (O) x R, where 

a) = ^ + (-A + A 4 ) _1 (^^(u)"^ - aMi*i(w)«i) . ^ - 1 j • 

Note that f|2 . 14[) holds if and only if \l/(u, Wi,ai) = (0,0). By computing the 
derivative of "J with respect to (i>, a) at the point (u,Wi,cti) in the direction 
(iD, a), we obtain a map $ : Hq (Q) x R — >• ffg(O) x R given by 

$(w,a) :=D t ,, Q ^I'(u, «i)(ui, a) 

= ftt> + (-A + A,)" 1 (\P\tj(u)u?w - a^iti(u)uf) , J u^wdx^j. 

If <&(w, a) = (0, 0), then we multiply the equation 

—Aw + XiW + \(3\tj(u)UjW = a[iiti(u)tq 

by w and obtain 

Jn 

So w = and then afj,iti(u)uf = in O. Since /ij > 0,tt(u) > and |iij|4 > 1/2, 
we see that a = 0. Hence $ is injective. 

On the other hand, for any (/, c) € Hq (Q) X R, let v\, U2 € -ffp (^) be solutions 
of the linear problems 

- Avi + XiVi + |/%(ff)uf«i = (-A + Xi)f, 

- Av 2 + XiV 2 + \/3\tj(u)v,jV2 = Hiti(u)v%. 

Since \ui\4 > 1/2, so ^ ^ and then J Q u^V2dx > 0. Let cxq = (c — 
J^u^vi dx)/ f n u?V2 dx, then $(wi +ao^2,ao) = (f> c )- Hence $ is surjective, 
that is, $ is a bijective map. This completes the proof. □ 
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Lemma 2.4. Assume that {«„ = u n ,2) '■ n> 1} C Aip is bounded in H 

and u n — u = (1*1,112) £ weakly in H. Then there exists w € H such that, 
up to a subsequence, w n := K{u n ) — > w strongly in H . 

Proof. Recall the definition of Mp in (1277)1 . we deduce from ((231) and (gTS} 
that there exists Co > independent of such that 

MilK'llf. + I/3|II m ?IIa- fo u i u 2 i 

* W = " a-INIl>a. V«6A«,. (2.18) 

Since u„ — 1 u = (1*1,1*2) € weakly in if, so up to a subsequence, 

Un,i ~> Ui strongly in L A {Vt). Then 

2 

,2 „,2 \ 1 ,, ,, I a\2 I I „,2„,2 ' 



lim I MlM2 - Ifl' (J^ <i< a J J = Mi^2 - |/?r (J^ «f«5 J > o, 

where the assumption u € .A/J^ is used. Hence we may assume that U(u n ) 
are uniformly bounded for any n > 1 and i = 1,2, and up to a subsequence, 
ti{un) ti > 0. Recall that w,^ = a n ^w n ^, where a n ^ and w n ^ are seen in 
|2T2l - |2~T3)) . By (j27T2l) we have ' 



n 



which implies that {w n ,i : n > 1} are bounded in Hq(Q). Up to a subsequence, 
we may assume that w n ^ — ¥ Wi weakly in H^(fl) and strongly in L 4 (£l). Then 
by (|2.12p and Holder inequality we get 

Vw„, l V(w„ il - Wi) dx + Xi / w nj i(w nj i - Wi) dx 

Jn 

P\tj(u n ) / U\ \,jW n ,i(w n ,i - Wi) + (iiti{u n ) / Un AWnj — W^dx-tO 



Jn Jn 
as n — > oo. Hence 

ll^n.illXi = / (VlS ri ,iVlOj + XiW^Wi) + o(l) = UtOiH^ + o(l), (2.19) 

Jo 

that is, w„.i — > Wi strongly in H^fl). Again by l|2.12p we know that Wi satisfies 

-Auii + XiWi + \0\tjUjWi = [iiUu\. 
Since |itj|4 = 1, so Wi ^ and then J„ ufiUi da; > 0, which implies that 

1 1 



lim a n i = lim 



jn U n,i™n,i Jn U i W i 

Therefore, w n ,i — Q. n ,i w n,i ~ * ctiWi Wi strongly in Hq(Q). □ 
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To continue our proof, we need to use vector genus introduced by [35J to 
define proper minimax energy levels. Recall (|2 . 1 6[) and (|2.7[) . as in [28] we 
consider the class of sets 

J 7 = {A C M : A is closed and cr* (u) G A V u G A, i = 1, 2}, 

and, for each A 6 T and k\, k 2 G N, the class of functions 

!2 fi-.A^- M fcs_1 continuous, "1 

/ = (ji, / 2 ) : A -> JjR*'- 1 : /^(fl)) = for each i, \ 

i=i /i (ctj- (5) ) for j ^ i J 

Here, we denote M° := {0}. Let us recall vector genus from [28] . 

Definition 2.1. (Vector genus, see 128]) Let A € T and take any k\,k 2 G N. 
We say that j(A) > (fci, /C2) if for every f £ i 7 ^^)^) i/iere exists u € A such 
that f{u) = (f 1 {u),f 2 {u)) = (0,0). We denote 

T (k lM ) : = {A e T : j(A) > {h,k 2 )}. 

Lemma 2.5. (see [281) With the previous notations, the following properties 
hold. 

(i) Take A x x A 2 C M and let r\i : S k ^ Y := {ir G R fei : |x| = 1} ->• A, &e a 
homeomorphism such that r/i(—x) = —r)i(x) for every x G S ,fci_1 , i = 1,2. 
T/ien Ai x A 2 G r( fcl ^ 2 ). 

(m) We have 77(A) G r( fcl,fe2 ) whenever A G r^ 01 ^ 2 ^ and a continuous map 
r) : A ^ A4 is such that rj o = a\ o 77, Vi = 1, 2. 

To obtain sign-changing solutions, as in many references such as [TT1 l4l |33]. 
we should use cones of positive functions. Precisely, we define 

2 

Vi :={u= («i,u 2 ) G-H" :«< > 0}, P := [J ('Pi U -Pi). (2.20) 

i=l 

Moreover, for <5 > we define Vs := {u E H : dist 4 (u, P) < 5}, where 

dist 4 (u,P) := min{dist 4 (ii l , Vi), dist 4 (u l , -Vi), z = 1,2}, (2.21) 
dist 4 (iti, ±Vi) := inf{|wi — u| 4 : t> G ±Vi}. 

Denote w ± := max{0,±u}, then it is easy to check that dist 4 (iij, ±Vi) — \uf\4. 

Lemma 2.6. Let ki,k 2 > 2. Then for any 5 < 2" 1 / 4 and any A G r( fcl < fc2 ) 
there holds A\V S ^ 0. 

Proof. Fix any A G r^ fcl,fc2 ^. Consider 
/ = (/ ll / 2 ):i4l il - 1 xK t '- 1 , /i(fl)= (JjuifuidxA--- ,oV (2.22) 
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Clearly / £ i 7 ^^) (A), so there exists u £ A such that f(u) = 0. Note that 
SeicM,we conclude that 

/ {v+fdx= I {urfdx = 1/2, fori = 1,2, 
Jn Jn 

that is, dist 4 (u, V) = 2" 1 / 4 , and so u £ A \ V s for every S < 2' 1 / 4 . □ 

Lemma 2.7. There exist A G r^ 1 ^' 2 ) and a positive constant c kl ' k2 £ N mde- 
pendent of (3 < suc/i £/ia£ sup^ < c kl ' k2 for any (3 < 0. 

Proof. Take nonempty open subsets Bi,-B 2 C f2 such that £?i n B2 = 0- Let 
Wk : 1 < fc < fci} C H^Bi) be linearly independent subsets, and define 

Ai := {u G span{^, ■ • • ,<^..} : |u| 4 = l}. 

Note that there exists an obvious odd homeomorphism from S^^ 1 to Aj. By 
Lemma [231 - (i) one has A := A\ x A 2 € r( fcl ' fc2 ). For any u — (111,112) € A, since 
G Hq(Bi), so wi • U2 = 0, which implies u £ Jvip and 

Jp{u) = J— "(HM!! + MiIMIa 2 )- 

Since all norms of a finite dimensional linear space are equivalent, so there exists 
Cki > such that H^Ha* < CfcJ^U = Cfci f° r an y M « € Hence there exists 
c ki,fca g p;} independent of /3 < such that sup A Jp(u) < c klM for any f3 < 0. □ 

For every fci, ki > 2 and < 6 < 2" 1 / 4 , we define 

4f 2 ■■= inf , , sup J^u), (2.23) 

where 

rjj fcl ' fc2) := I A e r( fel ' fe2 ) : sup J/? < c fel ' fc2 + l| . (2.24) 

Lemma [2771 yields p^* 1 '* 2 ^ ^ and so c k pf 2 is well defined. Moreover, 
c kiM < c fei,fe for every ^ < and 5 > 0. 

Recall that infx Jp > C\, so cj^f 2 > C\ > 0. We will prove that c k pf 2 is a 
critical value of Ep provided that 5 > is sufficiently small. As we will see in 
Remark O we can not replace K fcl ' fc2) by r( fcl < fc2 ) in the definition of c^f 2 . 

Lemma 2.8. For any sufficiently small 5 £ (0, 2 -1 / 4 ), there holds 

disU{K{u),V) < 5/2, yu£M, Jp(u) < c klM + 1, dist A (u,V) < 5. 
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Proof. Assume by contradiction that there exist S n — > and u n = (u n ,i, w n ,2) € 
M such that Jp(u n ) < c kl < k2 + l, dist 4 (n„,'P) < 5 n and Aist i {K{u n ),V) > S n /2. 
Without loss of generality we may assume that dist 4 (u n , V) = dist 4 (M„,i, Vi). 
Recall the definition of Jp , we see that u n € M.p and 

^ + 1 > Jp(u n ) > iMall^xllt+Mill^Hi 
4 ^ 2 -|/3| 2 (/n<i< 2 ) 

This implies that u n are uniformly bounded in H. Up to a subsequence, we may 
assume that u n — > u = {u\ : U2) weakly in H and strongly in L 4 (f2) x L 4 (i7). 
Hence |ui| 4 = 1 and u e M. Moreover, since (|2.8[) yields || a- — C > 0> 
where C is independent of n, so we deduce from f|2.25[) that 

Mi^2 - |/?| 2 f / w 2 u 2 ) = lim [ MlM2 - |/?| 2 f / iiL< 2 ) ) > 0, 



that is, u e A4^. Write K(u n ) — w n — {w n: i,w n ^) and = a n ^w n ^ as in 
the proof of Lemma \2. 41 Then by the proof of Lemma [2.4[ we see that tj(u n ) 
and a nj i are all uniformly bounded. Combining this with (|2.14p . we deduce that 



dist 4 (w n s,Ti)\w 7 a | 4 = K, ill < C / iV^ip + AiK jfrfi 



,- I - . /"< / I V - \ / pit i- . 

'a 

<C I (|V< 4 | 2 + A 1 (u;- 1 ) 2 + |/3|i 2 (u„)< 2 ( U ;- 1 ) 2 ) 



c 3 

IQ. 

< C (\l) 3 <l* < CK,ll4l^i|4 

Jn 

= Cdist 4 (u„,i,^i) 3 h,7,il 4 ^ C^l^n.iU- 

So dist 4 (iT('ii n ),'P) < dist 4 («?„,!, "Pi) < CS^ < S n /2 holds for n sufficiently 
large, which is a contradiction. This completes the proof. □ 

Now let us define a map 

V:M*p^H, V(u) :=u- K(u). 



We will prove that (y/ti(u)ui, \Jt2{u)U2) is a nontrivial solution of (|1.2j) . if 
u = (ui, U2) € Aip satisfies V(u) — 0. 

Lemma 2.9. Let u n = (w n ,i,u„.2) € .M/3 fee smc/i i/iai 

Jp{u n ) — ¥ c < 00 and V(u n ) strongly in H. 

Then up to a subsequence, there exists u £ Mp such that u n — > u strongly in H 
and V(u) = 0. 
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Proof. Without loss of generality we may assume that Jp(u n ) < c + 1 for all 
n > 1. Then by the proof of Lemma 12.81 up to a subsequence, we may assume 
that u n — 1 u = (ui, U2) G M.p weakly in H. By Lemma l2.4[ there exists w G H 
such that, up to a subsequence, w n := K(u n ) = {w n> \,w n ,i) w = (^1,^2) 
strongly in H. Recall V(u n ) — > 0, we get 



Vu n ,iV(«n,i - Wi) = / V(w„.i - iyi)V(u n ,j - Ui) + / VujiV(u„,j - u») 
Jn Jo 

+ / V(u„,i - w„,i)V(u n ,i - Ui) = o(l). 



Then similarly as (|2.19l) we see that u n — >• u strongly in H . By Lemma \2. 31 we 
have V(u) — lim^oc V(u n ) =0. □ 

Lemma 2.10. Recall C > in Then 

j' p (u)[V(u)} > C \\V(u)\\ 2 H , for any u G Mp. 

Proof. Fix any u = (ui, U2) G .M^, write w = K{u) = (wx, W2) as above, then 
V(u) = (ui — u>i, U2 — W2). By (|2.14p we have J n uf (ui — Wi) dx = 1 — 1 = 0. 
Then we deduce from (|23| - (|2~TU|) . (12TTI)) and (gZOj) that 

4(u)[t/(u)] 

= /Jj( u ) / (VujV(ui — tOi) + \iUi(ui - Wi) +ij(u)| J 0|u i (u i - Wi)uJ) dx 
i=i 

2 f 

>y]tj(rt) / (VuiV(ui — toi) + XiUi(ui - Wi) +tj(u)\/3\wi(ui - Wi)uf) dx 
i=i 

= y^Jj(u) / (VuiV(ui - Wi) + XiUi{ui - Wi) - VwiV(ui - Wi) 
i=i Jn 

- \iWi(ui - Wi) + oti(j,iti(u)uf(ui - Wi)) dx 

= V f \V{ui-Wi)\ 2 + \i\ui-Wi\ 2 dx> C \\V{u)\\ 2 H . 
i=i Jn 

This completes the proof. □ 

Lemma 2.11. There exists a unique global solution r] = (771,772) : [0,oo) x 
A4p — » i? /or i/ie initial value problem 

j t r)(t,u) = -V(r)(t,u)), r](0,u)=u£Mp. (2.26) 

Moreover, 

(i) 77(73,7!) eM^ /or anj/ £ > and u £ Aip. 
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(ii) Tj(t, <Ji(u)) = <Ji(r](t, u)) for any t > 0, u £ M.p and i = 1, 2. 
(Hi) For every Se Mp, the map t <-> Jp(rj(t,u)) is non-increasing, 
(iv) There exists Sq £ (0, 2" 1 / 4 ) such that, for every 5 < 5q, there holds 

r](t, u) e Vs whenever u £ Mp n Vs, Jp(u) < c klM + 1 and t > 0. 

Proof. Recall Lemma OH one has V(u) £ G 1 {M* p ,H). Since Mp C M* 
and Mp is open, so (12.26)) has a unique solution 77 : [0,T max ) x Mp —> H, 
where T max > is the maximal time such that T)(t, u) £ Mp for all t £ [0, T max ) 
(Note that V(-) is defined only on Mo). We should prove T max = +00 for any 
u £ Ad p. Fix any u — (u% , U2 ) £ Mp, one has 

— / r)i(t, uf dx = -4 f ra(t,ii) 3 (rii(t,u) - Ki(rj(t,u))) dx 



4-4 / rjii^U) 4 dx, V0<t<T n 
Jn 



that is 

d 



e 4t / ^ufdx-l 



dt 

Recall that Jo ^(0, u) 4 dx — J n u\ dx — 1, we see that 



= 0. 



/ rn(t, u) 4 dx ee 1 for all < t < T n 
Jn 



So i](t,u) £ M, that is i](t,u) £ M n = .M^ for all t £ [0,T max ). As- 
sume by contradiction that T max < +00, then either T](T lrLax ,u) £ M\ M*n or 
lim t ^T max \\r)(t, u)\\h = +°°- If r/(T max ,u) £ M\ M%, then the definition of 
Jp yields Jp(ij(T max ,u)) — +00. Since r](t,u) £ Mp for any t £ [0, T max ), we 
deduce from Lemma \2. 101 that 

f T — d 

Jp (ri(T max ,u)) = Jp(t](0,u)) + J —Jp(r](t,u)) dt 

= jp(u) ~ [ miX j'^t, u)W(v(t, m dt 



<Jp(u)-C \\V(r){t,u))f H dt< Jp(u) <+oo, (2.27) 

Jo 

a contradiction. So lim t _j.T max \\v(t: u)\\h = +00. Similarly as (|2.27[> . we see that 
Jp(r](t,u)) < Jp(u) < +00 for all t £ [0,T max ), and so 

' (Hl^iMlH + A*i|M*,d)llA a ) < Mv{t,u)) < Jp(u) < +00, 



which means that ||??(t, u)\\ 2 H are uniformly bounded for all [0,T max ), also a 
contradiction. Hence T max = +00 and (i), (Hi) hold. 
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By (|2.17l) we have V(ai(u)) = <Ji(V(u)). Then by the uniqueness of solutions 
of the initial value problem (|2.26p . it is easy to check that (n) holds. 

Finally, let So £ (0, 2~ 1 / 4 ) such that Lemma T2.8I holds for every S < 6q. For 
any u £ M.p with Jp(u) < c kl ' k2 + 1 and dist 4 (w, V) = 8 < So, since 

r)(t,u) = u + t—ri(Q,u) + oit) = u - tV (u) + o(t) = (1 - t)u + tK (u) + o(t) , 
at 

so we see from Lemma 12.81 that 

dist 4 (ri(t,u),V) = dist 4 ((l - t)u + tK{u) + o{t),V) 

< (1 - t)dist 4 (u, V) + idist 4 (if (u), V) + o(t) 

< (1 - t)S + tS/2 + o(t) < 5 

for t > sufficiently small. Hence (iv) holds. □ 

Proof of Theorem 11.11 

Step 1. Take any S £ (0,So). We prove that (|1.2I) has a sign-changing 
solution (ui,U2) £ H such that Ep{ui,U2) — cp'$ 2 - 

Write c'pf' 2 simply by c in this step. We claim that there exists a sequence 
{u n : n > 1} C M.p such that 

Jp(u n ) — > c, V(u n ) — » as n — ¥ oo, and dist 4 (u n ,7 3 ) > S, Vn £ N. (2.28) 

If (|2.28[) does not hold, there exists small e € (0, 1) such that 

\\V(u)\\ 2 H >e, Vu£ Mp, \Jp{u) -c\< 2e, dist 4 (u,7 ? ) > <5. 

Recall the definition of c in (|2.23|) , there exists A £ Y^t x such that 

sup < c + e. 

A\V S 

Since sup A < c klM + 1, so A C .M^. Then we can consider B = 77(2/60, A), 
where 77 is in Lemma \2. Ill and Co is in (|2.18p . By Lemma [2T5J-(ii) and Lemma 
I2.1U (m) we have B £ r^* 1 '* 3 ). Again by Lemma f2 . 1 II f m ) . we have sup B Jp < 
sup A Jp < c klM + 1, that is B £ rf lM) and so sup B \ Vs Jp > c. Then by 
Lemma [231 we can take u £ A such that 77(2/60, u) £ B\Vg and 

c-e< sup Jp - e < Jp(r)(2/C ,u)). 

B\Vs 

Since Jp(rj(t,u)) < Jp(u) < c kl - k - + 1 for any t > 0, Lemma l2.1U fa) yields 
r]{t,u) 7^5 for any t G [0, 2/ Co}- In particular, u <£Vb and so J/3 (it) < c + e. 
Then for any i £ [0,2/Co], we have 

c-e < Jp(r)(2/C ,u)) < Jp(j](t,u)) < Jp{u) <c + e, 
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which implies > e and 

±Jp( v (t,u)) = -j'p(r,{t,u))[V(r)(t,u))} < -C a \\V(^u))\\ 2 H < -C e 
for every t € [0, 2/C ]. Hence, 

/•2/Co 

c- £< Jp(r)(2/C ,ti)) < Jp(u) - / Coedt < c + e - 2e = c - e, 

Jo 

a contradiction. Therefore (|2.28|) holds, and by Lemma l23I up to a subsequence, 
there exists u — (m, u 2 ) <E A4p such that u n — > u strongly in H and V(u) = 0, 
J/3(w) = c = c'pf' 2 - Since dist 4 (u„, "P) > 5, so dist 4 (-u, T 3 ) > <5, which implies 
that both ui and u 2 are sign-changing. 

Since V(u) = 0, so u = K{u). Combining this with (|2.14[) - (l2.15p . we see 
that u satisfies 

I -Awi + A1U1 + \f3\t 2 {u)ului = , 
|-Aw 2 + A 2 «2 + |^|^i(w)wiU2 = a 2 ^ 2 t 2 (u)u 2 . 

Recall that \ui\4 = 1 and ti(u) satisfies (|2.4p . Multiply (|2.29p by and integrate 
over 57, we obtain that a± — a 2 — 1. Again by (|2.29p . we see that (u\,u 2 ) '■= 
(\/ti(u)ui, 1/^2(^)^2) is a sign-changing solution of the original problem (|1.2p . 
Moreover, ([2~6]l and (l2~TTj) yield 

Ep(ui,u 2 ) = Ji3(ui,u 2 ) = c k pf 2 . 

This completes the proof of Step 1. 

Step 2. We prove that (jl.2p has infinitely many sign-changing solutions 
(u nt i,u n . 2 ) such that 

||w n ,i|U°°(n) + ||w n ,2|U~(n) +00 as n y -(-co. (2.30) 

It suffices to prove that 

lim inf c^ 2 = +00. (2.31) 

Assume by contradiction that there exist fc™ ~~ > °°! G (0> 2~ 5 / 4 ] and a 

k n /Co 

positive constant C such that c^g < C for every n € N. Then there exists 
A„ € T ( p" M) such that 

sup J/3<C + 1, VneN. 

Let {iy9fc}A; C H^ip.) be the sequence of eigenfunctions of (— A,Hq(H)) as- 
sociated to the eigenvalues {Ak}k, then — > +00 as fc — >• +00. Define maps 
9 n = (9i,92) ■ A n -> K fc " _1 x R^- 1 by 

flfW := ( / - , / Vk?-2«x, / K|V ) , (2.32) 

vn Jn in / 
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92 (u) ■= / <PlVQ,~- , / <Pk 2 -2U 2 , / \U 2 \ 3 U 2 

\Jn Jn Jn 

Then g n £ F( k ^ k ^{A n ) and so there exists u n — (ui,u r 2 l ) £ A„ such that 
g n (u n ) = 0. As in the proof of Lemma 12. 6[ firstly this means u n £ A„ \ 
"P^ and so Jp(u n ) < C + 1 for every n G N. Secondly, we have u™ € 
span{y>i, • • • , and so f Q \Vitf\ 2 > A-kf-i J Q \v%\ 2 . Recall that 

C* + l>^(^)>-^||<|| 4 Al , 

we see that u™ are uniformly bounded in i?g(f2). Up to a subsequence, we may 
assume that it™ — > u\ weakly in i?o(fi) and strongly in L 2 (£l) n L 4 (f2). Since 

<| 2 < t^— / l Vu ™ I 2 as n +°°' 



A fe? - 



so Ui = 0. On the other hand, u n £ A n c A4 yields |u™|4 = 1 for any n, so 
\u1\4 = 1, a contradiction. Therefore (|2.31[) holds and so (11.21) has infinitely 
many sign-changing solutions (u n ,i,u n ^) such that Ep{u n ,x,u n ,2) — > +°o as 
n — > oo. By standard elliptic regularity theory, we see that u n ,% £ L°°(Q). Since 

4Ep(u n ,l, U n ,2) = Hl\u n ,\\\ + ^2\u n ,2\t ~ 2|/3| / u\ ^ 2 

Jn 

< Ml|fi|||«r l) l|ll«(£2) +M2|^|||Ur 1 ,2|lloc (a) , 

so (|2.30|) holds. Here |fi| denotes the Lebesgue measure of il. This completes 
the proof. □ 

Remark 2.2. Ao£e f/iaf if A £ T^ 1 ^ \ Y^ lM) , then we can not consider the 
set 77(2/60, A) in the proof of Theorem \l.l\ because rj(t, •) can not be defined on 
the whole M for any t > and so 77(2/6*0, A) is not well defined. Hence we can 
not replace p^ 1 ^ 2 ' by p(*iife) i n tfag definition of cj 1 ^ 2 . Define 

ff M) := I A £ r( fel ^> : supJ^ < +00 j . 

Then for any A £ p^* 1 '* 2 ' ; i/ie set B = r)(2/Co, A) is well defined. Take u £ A 
such that 77(2/Co, u) £ B\Vs as in the proof of Theorem ] 1. 11 Then, since we do 
not know whether Jp(u) < c kl ' k2 + 1 holds or not, it seems impossible for us to 
prove u (jL Vs, which plays a crucial role in the proof of Theorem \l.l\ Therefore 
we can not replace p^ 1,fe2 ) by p^ fc i' fe 2) - n ^ g d e j[ n {ii on of c^f 2 either. 



3 Proof of Theorem O 

In this section, let k\ — k 2 — 2 and take 5 > small enough such that c^\ is a 
critical value of Ep. Write c 2 '^ by c for simplicity. By the proof of Theorem ll.il 
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we see that (|1.2p has a sign-changing solution U = (U\ 1 U2) such that 

Ep(U) = c < c 2 ' 2 . (3.1) 

We will prove that U is a least energy sign-changing solution. To do this, let us 
define 



c := inf Eb(u), 



(3.2) 



where 



Afp := I it = (ui, U2) G H : both u\ and u 2 change sign, 

E'p(ti)(uf,0)=Q, E'p{u){Q,uf) = 0}. (3.3) 

Then any sign-changing solutions belong to Afp. In particular, U € A(a and so 
2 < Ep(U) — c < c 2,2 . To prove the opposite inequality c > c, we need the 
following lemma. 

Lemma 3.1. Let u — [u±, u^) £ Afp, then 

Ep(ui,u 2 ) = sup Ep (yftuf ■ 
tf,tf>o V 

Proof. Note that E'Ju)(uf , 0) = and E'(u)(0, uf) = yield 



Ml i V ^2 u 1 



t 2 u 2 



(3.4) 



Hi\ut\i = \\ut\\i+\P\ I \uf\ 2 ul i=l,2. 



Then 



2|/3| 



— y ^ 



^2 M 2 V ^2 U 2 



=2\(3\ttt+ / ( U +) 2 ( U +) 2 + 2|/3|^ 2 - / K) 2 ( U2 -) 2 
Jo Jo 

+ 2|/3|^+ / (%) 2 «) 2 + 2|/3|fr*2 / K) 2 K) 



<|/3|[(tf) 2 + (4) 2 ] «) 2 (4) 2 + l/3|[(^) 2 + fe-) 2 ] «) 2 (« 2 -) 2 



+ m [(o 2 + (4) 2 ] / («r ) 2 (4) 2 + m [(o 2 + fe-) 2 ] / K) 2 (% ) 2 
=i/?i(tf) 2 / «)M + i/3|(*r) 2 / (%) 2 « 2 

Jo Jo 

+ \mtf [ u\{uif + \m^f I «?(%) 2 

Jo Jo 

=(4)VKIS + (tr) 2 /iii% \i + (4) 2 ^\u+\i + (t 2 -) 2 /x 2 |% \i 
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- (tffwu+wi (tr) 2 n% ill - (4) 2 k + ii 2 a 2 - (*2 ) 2 k ill- 

Hence for any tf , tf > we deduce that 



*1 M l i \/^t u 2 



t 2 U 2 



= 2*i Kilt + 2^ l|u 



! + 2*2 ll M 2 IIA 2 



+ 2*2 ll U 2 IIA 2 



<-±Ci 

i=l v 



*2 U 2 V *2 U 2 



H II Ai 



Eft 



^2 



Till 



1 



<4 (Kill + 11% 111 + Kill + KIIaJ = Ep{u u u 2 ). 



Letting {t{ ,t 1 ,t 2 ,t 2 ) = (1, 1, 1, 1), we completes the proof. □ 
Lemma 3.2. c = c and so U is a least energy sign-changing solution of 
Proof. Take any u = (ui,u 2 ) € Afp such that Ep(u) < c 2 ' 2 + 1. We define 
A := Ax x A 2 ; := {u £ span{w+,ur} : | u | 4 = 1}, 

As in the proof of Lemma [2.71 one has A £ I^ 2,2 ). For any v — (i>i,i>2) £ A, 
there exist bi,di £ K such that = btuf + dtu~ . Then by (|2.11[) and Lemma 
I3.1lwe have 



Jp(v) = SUp Ep(y/tiVl, Vt2V 2 ) 
ti,t 2 >0 



sup Ep(\/ii(biui + d\u l ), vT^fewJ + ^2M 2 )) 

*1,*2>0 



= sup ^(vtil^ilui - V*i|diK , V*2|&2| u 2 - V^lc^K ) 

tl,*2>0 

< Ep(u 1 ,u 2 ), 

that is, sup^ J/3 < Ep(u) < c 2,2 + 1 and so A £ T^' 2 \ which implies 

c = c 2,2 < sup Jp(v) < Ep(u), \fu£ Afp with Ep(u) < c 2,2 + 1. 

v£A\V s 

Hence c < c, that is, c = c = Ep(U). Since any sign-changing solutions belong 
to Afp, so U = (Ui, U2) is a least energy sign-changing solution of (|1.2j) . □ 

To continue our proof, we need a classical result by Miranda. 
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Lemma 3.3. (see 122]) Consider a rectangle R = ni[ a »'^] c ^ s an d a con ~ 
tinuous function $ : R ->• W , $ = ($!,-•• , $ s ). If $i\ Xi =a t > > $i| ai =6i 
Ziolds /or every i, then <f> /ias a zero inside R. 

Lemma 3.4. Soi/i ?7i and 6^2 has exactly two nodal domains. 

Proof. Since U\,U-2 both change sign, so both U\ and U2 have at least 
two nodal domains. Assume by contradiction that U% has at least three nodal 
domains fii, f2 2 and f^. Without loss of generality, we assume that U\ > on 
U fi 2 . Define 



,+ — 



where 



Xn(x) :-- 



1, ie!l, 

0, x <E M. N \ ft. 



Then uf,u 3 € H%(Q) \ {0}. By E' p (U)(uf,0) = and E' p (U)(0,U^) = 



have 



±|4 



I^IIL 



Let 



,±112 



(3.5) 
(3.6) 



It/. 



a := — mm ■ 
2 



2" II A 2 



IK III ; 

Misfit' t*2\ U 2 



> 0. 



From dSSJ-dSU) one has a < 1/2. For any > 1, we define $ = (/ x + , /f, / 2 + , / 2 ") : 
[a, o] 4 -» R 4 by 



/i ± (*i",*r.*a>*i') : = ikfllL 11 + 1 

ff(ti,ti,t+,q) := 11^11^-^1^11 + 
Then for any € [a, 6] 4 , 



(«f) 2 



4^2 + 



n U T ~ v *i u i 



±\2 



/flt±=„ > Infill - am\uf\i > -K|| 2 Al > 0, 
f?\tf= a > \\U£\\l - a» 2 \U?\i > ±\\Uf\\l 2 > 0. 



Moreover, by (|3.5j) we have 



11^111-^11^11 + 



Hf=b 



±\2 



(«f) 



TO" 
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htWl-bfxxiufii + iPi / (ut) 2 (t+(u+) 2 + t;(u 2 -f) 



n 



= (l-b)\\uf\\t 1 <0, (3.7) 
Similarly, by (|3.6[) we have 



f£\t±= b < 11^111-^21^11 + 61/31 / Ul{Uff = (1- b)\\U±\\l 2 <0. (3.8) 
Then by Lemma |3~31 there exists (if, ff, ij, G [a, 6] 4 such that 

/l (^1*) ^1 i ^2") ^2 ) = 0, /2 ^1 ) ^2") £j ) =0. 

This implies that 



Remark that (|3.7|) and (|3.8[) hold for any 6 > 1, so we obtain that 
tt < 1, £f < !) *2 ^ !) *2 < !• 

Hence 



c < E fi < - v*r«r> v*2 ^ - V*2 ^: 
= J (^IKIIL +tiwni +«2 II^-IIL) 

< \ (11411a, + IMIL + IMI Al + ||£tfHL + II^IIL) 

< J (H^iIlL + IINlL) = EpiUM = c, 

a contradiction. Hence U\ has exactly two nodal domains. Similarly, Z7 2 has 
exactly two nodal domains. □ 

Theorem 11.21 follows directly from Lemmas 13.21 and 13.41 □ 

4 Proof of Theorem 11.31 

The following arguments are similar to those in Sections 2-3 with some impor- 
tant modifications. Here, although some definitions are slight different from 
those in Section 2, we will use the same notations as in Section 2 for conve- 
nience. To obtain semi-nodal solutions (ui,ti2) such that u\ changes sign and 
U2 is positive, we consider the following functional 



2„,2 
21 

Q, 
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and modify the definition of H by 

H := {(u u u 2 ) € H : u x ? 0, u+ ^ 0}. 
Then by similar proofs as in Section 2, we have the following lemmas. 
Lemma 4.1. For any (u 1 ,u 2 ) G H , if \f3\ 2 (f n u\ u 2 ) 2 > ^i^lwilllw^ll; then 

sup Ep(^/t[ui, \fU2u2) = +00. 

t!,t 2 >0 

Lemma 4.2. For any u= (1*1,1*2) G H, if 

u\ul) < MiMaM*!^^, (4.1) 



then system 



||w 2 |Ia 2 =t2Li2\u+\i-t 1 \(3\f n u 2 1 v 



has a unique solution 



' _ M2| M2 + |^|kl|| 2 Al +|/3||| M 2|| 2 A2 f n ulul 

1[U> MiM|ui|i|u+|J-|0| 2 (/ n «?«!)% 

1 / — *\ Mi|"il4ll"2||I 2 + |^|||m||i 1 / n «i«2 q 

2[U> ~ MlM2|«llll« 2 + ll-|^| 2 (/ s ,« 2 «l) 2 > U ' 



Moreover, 



sup £73 (a/FTui, Vh.u 2 ) = E fj [yjti{u)ui, y/t 2 (u)u2 

*l,t 2 >0 v 

= i M2h4l!hi|| 4 Al +2|/?ih 1 ||^|| M2 f A2 / n ^| + Mi|u 1 ||h 2 |i A2 

4 MiM2|«i|t|« 2 + |t-|/3| 2 (/ n ^l) 2 

and (ti(u),t2(u)) is the unique maximum point of Ep(-\ftiU\, y/t^u^)- 

Now, we modify the definitions of M*, M*p, M*p* , M and Mp by 

X*:={uei7 : |ui| 4 > 1/2, |uJ| 4 >l/2}; 
.M^ := {u £ M* : u satisfies (l4"7Tj) } ; 

X;*:=|mgX* : - |/3| 2 f / u\ul] >o| 



(4.2) 



(4.3) 



(4.4) 



M:={t?Gff : |«iU = l, |«JU = 1}, A<p:=MnM;, (4.5) 
and define a new functional J/3 : A4* — » (0, +00] as in Section 2 by 

\ /.2 11m 11^+21^111^1 || A J n ^^+Ml|l"2|ll 2 - f ^, A >.* 

Jb{u):={* MiM2-|/3| 2 (/ ri « 2 ^) 2 iiutyvi^, 

-00 if ue .A/T \A4" 
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Then Jp £ C(M*,(0,+oo]), inf^* Jfs > C\ > where C\ independent of 
13 < 0, J p £ ^{Mf, (0,+oo)) and (i2~9l) - (j2~T0)) hold for any u £ A4p and 
tp, ij; £ Hq(CI). Note that Lemmas I4TT1 and I4T21 yield 

Jp(ui,u 2 )= sup E/3 (^/hux, y/%u 2 ) , V(ui,u 2 ) £ M. (4.6) 

ti,t 2 >0 

For any u = (iti, u 2 ) £ -M*a, let w)j £ Hq (f2), i = 1, 2, be the unique solutions 
of the following linear problem 



J -Au)i + Xiwi + \(3\t 2 (u)ulwi = fiiti(u)ul, wi £ Hq(Q), 
\-Aw 2 + X 2 w 2 + \p\ti(u)ulw 2 = n 2 t 2 (u)(u+) 3 , w 2 £ H^(n). 

As in Section 2, we define 



(4.7) 



Wi = otiWi, where eti = - r — ^r- > 0, a 2 = , > 0. (4.8) 

Then (w 1 ,w 2 ) is the unique solution of the problem 

—Awi + Xiwi + \(3\t 2 {u)u 2 wi — ai^\ti{u)u\ , w\ £ H^(Cl), 
-Aw 2 + \ 2 w 2 + \(3\ti(u)v%W2 = a 2 fi 2 t 2 (u)(ut) 3 , w 2 £ H^(Q), (4.9) 
Jo ufwi dx = 1, Jq( u 2 ) 3w 2 dx = 1. 

As in Section 2, the operator K = (Ki,K 2 ) : A4*^ — > H is defined as K{u) := 
w = (w\,w 2 ), and similar arguments as Lemma 12.31 yield K £ C l {M.%, H). 
Since u n — > u in L A (Vt) implies u+ — > u + in L 4 (f2), so Lemma 12.41 and its proof 
with obvious modifications also hold for this new K defined here. Note that 

K{a x {u)) = a 1 (K(u)). (4.10) 

Remark that (14. 10)) only holds for o\ and in the sequel we only use <j\ . Consider 

T = {A c M : A is closed and <t\(u) £ A Vie A}, 

and, for each A £ T and k\ > 2, the class of functions 

F (kul) (A) = {f:A^ R fe i-i : / continuous and /(oi(tZ)) = -f{u)} . 



Definition 4.1. (Modified vector genus, slightly different from Definition \2. 1\) 
Let A £ J- and take any k\ £ N with ki > 2. We say that j(A) > (fci, 1) if for 
every f £ Fr^^fA) there exists u £ A such that f(u) = 0. We denote 

r (ki,i) ~{A£F:1(A)>(k 1 ,l)}. 

Lemma 4.3. With the previous notations, the following properties hold. 

(i) Take A := A\ x A 2 C M and let rj : S kl ~ 1 — > A\ be a homeomorphism 
such that r){—x) = —T)(x) for every x £ S kl ~ 1 . Then A £ r( fcl:1 ). 
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(ii) We have r](A) G r^ 1 ' 1 ) whenever A 6 r^ 1 ' 1 -* and a continuous map r\ : 
A — > M is such that 77 o a\ = a% o 77. 

Proof. The conclusion (ii) is trivial, we only prove (i). Fix any / £ F^ kl l - ) (A) 
and take any u 2 € A 2 . Define y> : S" fcl " 1 -> M fe i-i by p(x) := f(rj(x),u 2 ). Then 
is continuous and <£>(— x) = — </?(x). So by Borsuk-Ulam Theorem, there exists 
xo G S' fel_1 such that (f(xo) = 0, that is /(f7(xo), U2) — 0. Hence j(A) > (kx, 1) 
and A G r^ 51 ' 1 ). □ 



Now we modify the definitions of V and dist 4 (u,7 7 ) in (|2T20]) - (|2T2T|) by 
V := Vi U -Vi, dist 4 (w,7') := min{dist 4 (Mi, Pi), dist 4 (ui, -Pi)}. (4.11) 
Under this new definition, u\ changes sign if dist 4 (u, V) > 0. 

Lemma 4.4. Let k\ > 2. Then for any 5 < 2~ 1//4 and any A G T^ 1 ' 1 ) there 
holds A\V S ^ 0. 

Proof. Fix any A G r^ 1 ' 1 ). Recall the map fx : A -> R fcl_1 defined in (|2~22"1) . 
Clearly /1 G ^(^^(A), so there exists Sei such that fi(u) = 0, which means 
Jn( u i~) 4 = Jtt( u D 4 = V 2 , that is > dist 4 (u,P) = 2" 1 / 4 , so u € A \ Vs for every 
(5<2~ 1 / 4 . □ 

Lemma 4.5. Le£ fei > 2. T/iere exist yl G T^ 1 ' 1 ' and a constant c kl - x G N 
independent of /? < smc/i i/iai sup^ < c fc1,1 /or anj/ /3 < 0. 

Proof. Let Bi and : 1 < k < A;,-} C H^Bi) be in the proof of Lemma 12. 71 
Define 

Ai := {u G span{<^,--- : \u\ 4 = l}, A 2 = {C \<f%\ : C = 1/|^| 4 } . 

Then by Lemma [4. 31 - (i) one has A := Ax x A 2 e T^ 1 ' 1 ^. The rest of the proof 
is the same as Lemma [221 □ 



For every k\>2 and < 5 < 2 1 / 4 , we define 
fci, 



ck 0,s : = inf SIl P Jp{u), 



where the definition of pi* 1 ' is the same as (|2.24|) . Then Lemma [4.51 yields 
p(fci,i) _^ go fci^i j g we jj jjgf^jjgj^ Moreover, c^f < c fcl:1 for any fj < 
and (5 > 0. Under the new definitions (|4. 1 1 [) . it is easy to see that Lemma 
12.81 also holds here. Now as in Section 2, we define a map V : M.% — > H by 
V(u) := u — K(u). Then Lemma [231 also holds here. Recall from (|4.5|) and (|4.9|) 
that 

/ (uj ) 3 (m 2 - w 2 ) dx = 1 - 1 = 0, V u = (ui,u 2 ) G Mp. 
Jn 

Then by similar arguments, we see that Lemma [2.101 also holds here. 
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Lemma 4.6. There exists a unique global solution rj — (771, 772) : [0, 00) xM^ -> 
H for the initial value problem 

j t r)(t,u) = -V(<r](t,u)), r)(0,u) =ueMp. (4.12) 

Moreover, conclusions (i), (in) and (iv) of Lemma \2.11\ also hold here, and 
r)(t, &i(u)) = ai(r)(t, u)) for any t > and u G M.p. 

Proof. Recall V{u) G C l (M*^H), (j4TT2|) has a unique solution 7? : [0,T max ) x 
A4/3 —> H, where T max > is the maximal time such that rj(t,it) £ M% for all 
t G [0, T max ). Fix any u = (ui,u 2 ) E M.p, we deduce from (I4.12|) that 

-/ ( m (t,u)+f dx = -A ( (m{t,u)+) 3 (77 2 (i,u)™^2 (v(t,u)))dx 

•J Vt J Cl 



(ri2(t,u) + ) 4 dx, V0<t<T n 



that is 

~dt 



e 4 ' 



^Jj m (t,u)+) 4 dx-1 



= 0. 



Since J n (772(0, u) + ) da; = J n (uJ) 4 dx = 1, so J n (rj2(t,u) + ) dx = 1 for all 
< t < T max . Recalling (|4.10l) . the rest of the proof is similar to Lemma r2.11I D 

Proof of Theorem [TT3l 

Step 1. Fix any k\ > 2. We prove that c^ 1 is a sign-changing critical value 
of Ep for 8 > small. 

By similar arguments as Step 1 in the proof of Theorem ll.il for small 5 > 0, 
there exists u = (tti, U2) G .M^ such that 

Jp(u) = Cgf , V(u) = Q and dist^w, P) > <5. 

Then u\ changes sign. Since V(u) = 0, so u — K(u). Combining this with (|4.9p , 
we see that u satisfies 

j-Aui + X1U1 + \^\t2(u)ulu! = a 1 ^ 1 t 1 (u)ul, 
1 — + A2U2 + |jS|*i(tt)«iU2 = a 2 H2t2(u)(ut ) 3 - 

Since |ui I4 = 1, Ittg U = 1 an d satisfies (|4.2|) . so a\ — a 2 = 1. Multiply 
the second equation of (|4.13[) by 77^ and integrate over O, we get ||^ = 0, 
so W2 > 0. By the strong maximum principle, u 2 > in f2. Hence (ui,U2) := 
(\/ii(u)ui, yjt2(u)u2) is a semi- nodal solution of the original problem (11.2[) with 
t2i sign-changing and U2 positive. Moreover, (|4.4j) and (|4.6j) yield 

Epiui,^) = Ep(ui,v,2) = Jfj{ui,u 2 ) = c k p f < c* 1 ' 1 . 
Step 2. We prove that ()1.2j) has infinitely many semi-nodal solutions. 
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Assume by contradiction that there exist fe" — > oo, S n £ (0, 2 5 / 4 ] and 

k n 1 

a positive constant C such that Cg 1 ^ < C for every neN. Then there exists 
A n e T^ 1 such that sup^ n yp 5 Jp < C + l for any neN. Let {(y5fc}fc C i?o (^) 
be in the proof of Theorem 11.11 and recall the map <?" : A n — > M fc " _1 defined 
in (|2.32j) . Then € F^n^^An). By the same arguments as in the proof 
of Theorem 11.11 we get a contradiction. Therefore, (|1.2j) has infinitely many 
semi-nodal solutions {u n = (u ni i,u n ^)}n>2 which satisfy 

(1) u n> i changes sign and u n ^. is positive; 

(2) Ep(u ni i,Un,2) = c n ^ Sn < c 11 ' 1 for some < S n < 2" 1 / 4 . Moreover, 

||Wn,l||i«.(n) + ||ttn,2||z,~(Q) -> +oo as n4 +oo. 

Step 3. We prove that u n> \ has at most n nodal domains. 
Assume that u n> \ has at least n + 1 nodal domains fifc, 1 < fc < n + 1, then 
u n ,iXn k S Hq(Q). For 1 < fc < n, we see from Ep(u n )(u n> iXn k , 0) = that 

MiK.iXnJt = Hun.ixnJI^ + |/?| / K,iXnJ 2 i4. 2 > k = l,---,n. (4.14) 

Jo 

Similarly E'g(u n )(0, u n ^) = yields 




Then, similarly as Lemma |3. II we have 




fe=l 

Recall that u n .2 is positive, so for ii, • • • , t n , s > 0, 




27 



1 n 1 

< i Ell W «,^J| 2 Al + i hn, 2 || 2 A2 - (4-15) 
fe=l 

Now we define 

A := Ax x {Cw„, 2 : C = 1/K, 2 | 4 } ; 

Ai := {u £ span{u„ 4 xrii, • ■ • ,"n,iXn„} ■ |«U = l}- 

Then Lemma HT3}(i) yields A € r^™' 1 ), and similarly as Lemma T3.21 we deduce 
from (HH) and (|4~T5l) that 

1 " 1 1 

sup J/3 < jJ^W^iXnJ^ + 7lK,a||A a < 7 (IKiIIa, + IKalliJ 
A fc=i 

= E f3 (u n ) = c p ' Sn < c , 
and so A G rl™' 1 ^, which implies 

1 " 1 

Cfl'L < sup J^j < - V ||u n ,ixnJ| Al + tIK,2||a 2 < Ef)(u n ), (4.16) 

A W*n 4 fcl 4 " 

a contradiction. Hence has at most n nodal domains. In particular, U2,i 
has exactly two nodal domains. 

Step 4. We prove that (it2,i, 1*2,2) has the least energy among all nontrivial 
solutions whose first component changes sign. 

By similar arguments as in Section 3, we can prove that 



2.1 



= ^ inf E fi (u) = ^ inf E p (u), (4.17) 



where 

■A/2,1,0 := |w = (ttij M2) € H : Ui changes sign and u 2 > 0, 112 7^ 0, 



^(u)(u±,0) = 0, ^(fi)(0,tta)=0}. 



Let u — (ui,U2) be any a nontrivial solution of (|1.2[) with U\ sign-changing. 
Without loss of generality we assume 7^ 0. Then by a similar argument as 
Lemma T3.41 there exists tf,^ £ (0, 1] such that 



and so 



H U l - V t l U l ) V *2 U 2 G -^2,1,0, 



1 



< i (lkillA 1 + IKIIt.) < ^(fi). 

Hence (1*2,1, ^2,2) has the least energy among all nontrivial solutions whose first 
component changes sign. This completes the proof of Theorem 11.31 □ 
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5 Asymptotic behaviors and phase seperation 

In this section, we study the limit behavior of solutions obtained above as /3 — ¥ 
-co. Fix any k±, &2 G N such that k\ > 2 and k% > 1. Then by the arguments 
in above sections we know that, for any /3 < 0, there exists Sp € (0, 2~ 1 ' 4 ) and 
u,3 = (u/3,i, 14,3,2) G -ff such that up is a nontrivial solution (either sign-changing 
or semi-trivial) of (ll.2j) with 

<c fc1 ^ <+oo. 

Here c kl ' k ' 2 is seen in Lemmas 12.71 and 14.51 Recall that 

E p{up) = \ (II^.iIIa! + II u /3,2|Ia 2 ) i 

we see that up are uniformly bounded in H. On the other hand, by Kato's 
inequality (see [T5]) we have 

AM > Au 0ii ■ in (itf («))'■ 

\ u P,i\ 

Recall that (3 < 0, then it is easy to check that 

-Alltel + < Hilup^l 3 , \u0 ti \ e Hq(Q). 

Hence by standard Moser iteration, we see that up^ are uniformly bounded in 
L°°(il) for any (3 < and i = 1,2. Moreover, by elliptic regularity theory it 
holds that up t i g C(£!)nC 2 (£l). The main result of this section is following. 

Theorem 5.1. There exists a vector Lipschitz function i*oo = (uoo,i, ^00,2) € H 
such that, up to a subsequence, 

(1) up.i — ¥ Uoo.i in Hq(Q) n C°' a (fl) for every < a < 1 as (3 — >• -00; 

(2) -Amqo^ + AiUoo.j = * n °P erl se< { u oo,i ^ 0}; 

(3) 1*00,1 • iioo.2 = and |/3| w| x Up 2 dx — » as /3 — > —00; 

(4) ifk\,k 2 > 2, thenUoo^ changes sign fori — 1,2. Moreover, ifki = fc 2 = 2 ; 
i/ien {ifoo.i 7^ 0} /*as exactly two connected components, and is a least 
energy sign-changing solution of 

- Au + X lU = fi.u 3 , u e -ffoC-iAW 7^ 0}) (5.1) 

/or i= 1,2; 

(5) if ki > 2 and k 2 = 1, t/ien U00.1 changes sign, {"Uoo^i 7^ 0} has at most k\ 
connected components and Uoo,2 is positive in {^00,2 7^ 0}. Moreover, if 
(^1,^2) = (2,1), then {1*00,1 7^ 0} has exactly two connected components, 
1*00,1 is a least energy sign-changing solution of \5.1\) fori — 1, {uoo,2 7^ 0} 
is connected, and 1*00,2 is a least energy solution of H5.1]) for i = 2. 
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Proof. Since upj are uniformly bounded in L°°(f2) for any f3 < and i = 1,2, 
then (1) — (3) follows from [25l Theorems 1.1 and 1.2]. Remark that, although in 
[2"5] the results are stated for nonnegative solutions, they also hold for solutions 
with no sign-restrictions; all arguments there can be adapted with little extra 
effort to this more general case, working with the positive and negative parts of 
a solution. This fact was pointed out in the proof of [28; Theorem 4.3]. 

It remains to prove (4) — (5). First we consider the case ki, k 2 > 2. Since 
up,i £ C(0) n C 2 (f2) and upj changes sign, so there exists Xg i £fl such that 

u P,i( x B i) = max-up i(x) > and ua^XgA = min up Ax) < 0, i = l,2. 

Then Au^^a;^) < 0. Since up satisfies (|1.2p and /? < 0, so AiU^^ai^ .) < 
fJ-iUp^Xp^), which implies 

u P,i( x ti) = vxaxup t i(x) > y/Xi/fii, V/3 < 0. 

Similarly, 

up,i{xg ,) = vamup fi (x) < -y/h/Tk, V/3 < 0. 

Combining these with (1), we see that Uoc^ changes sign, and so {itoo,i 7^ 0} has 
at least two connected components. 

Now we let (ki, k 2 ) = (2, 2). Assume by contradiction that {^00,1 7^ 0} has 
at least three connected components fii, ^2 and ^3. Without loss of generality, 
we assume that itoo,i > on f2i U O2 . As in the proof of Lemma T3.4[ we define 
uf := xOiUoo.i, u^ := x^Uoo^i and u 3 := xo 3 Uoo,i, then ||iioo,i|lAi > H^iH^ + 
llu^H^. By Theorem l5.H -(2 > ) and (|3.3[) it is easy to see that (u^— u^[, u,-^, ->) € Afp 
for all (3 < 0, so 

7 (HuocxIIa! + II w oo. 2 ||a 2 ) = I™ - (||^,l|| Al + fellL) 

= lim Ep(up A ,up, 2 ) = lim 

p— >■ — OO p— ^ — OO ^' p 

< lim Es(nt - % ,1*00,2) 

= J (KIIL + IMIL + ll^iiy, (5-2) 

a contradiction. Hence {uoo,i 7^ 0} has exactly two connected components. If 
Ui G -ffo({uoo,i 7^ 0}) is any sign-changing solutions of (15.11) . then (u±, 1*00,2), 
(uoo,i, u 2 ) € A(a for all /3 < 0, so similarly as (|5.2p we see that ||«/.oo,*|| a — II w » III 
that is, Woo,i has the least energy among all sign-changing solutions of (|5.1[) . 
Hence itoo.i is a least energy sign-changing solution of (|5.1I) . and (4) holds. 

Now we consider k\ > 2 and fc 2 = 1. Then itoo.i changes sign as above. 
Since 11,3,2 is positive, so Uoo,2 > in {uoo,2 7^ 0}. Define 

■A4i,i,/9 := jtZ= (1*1,112) £ H : ui changes sign and has at least k\ nodal 
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domains fifc(l < k < fci), U2 7^ 0, U2 > 0, 



E' fi (u)(u lX n k ,0) = 0, Vl<fc<fci, ££(tZ)(0,u a ) = o} 



Then the same arguments as (|4.14l) - (|4.16[) yield that 



If {itoo.i 7^ 0} has at least k\ + 1 connected components < fc < fci + 1), 

then 



Similarly as (|5.2p we get a contradiction. Hence, {u^i 7^ 0} has at most fci 
connected components. 

If (fci,fc2) = (2,1), then {uoo^i 7^ 0} has exactly two connected compo- 
nents. If {1*00,2 7^ 0} has at least two connected components Sli and Vt>2, then 
(1*00,1, ) S -^2,i,/3 for all j3 < 0, and similarly as (|5.2p we get a contra- 

diction. Hence {u^ ^ 7^ 0} is connected. Finally, similarly as above, we can 
prove that is a least energy sign-changing solution of (|5.1[) for i = 1, and 
"00,2 is a least energy solution of (|5.1[) for i = 2. This finish the proof. □ 

6 The entire space case 

In this final section, we extend some results above to the case where SI = M. N . 
That is, we consider the following elliptic system in the entire space 



By giving some modifications to arguments in Sections 2-4 and introducing some 
different ideas and technicalities, we can prove the following results. 

Theorem 6.1. Let N = 2,3, Ai,A 2 , Mi>M2 > and f3 < 0. Then fP]) has 
infinitely many radially symmetric sign-changing solutions, including a special 
(111,112) such that both u\ and U2 have exactly two nodal domains and (u\,U2) 
has the least energy among all radially symmetric sign-changing solutions. 

Theorem 6.2. Let assumptions in Theorem \6.1\ hold. Then \6. 1\) has infinitely 
many radially symmetric semi-nodal solutions {(u nt i,u n ,2)}n>2 such that 

(1) Un,i changes sign and u n ^. is positive; 

(2) u 7lt i has at most n nodal domains. In particular, U2.1 has exactly two 
nodal domains, and (1*2,1,^2,2) has the least energy among all nontrivial 
radially symmetric solutions whose first component changes sign. 




V /3 < 0. 




(6.1) 
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Remark 6.1. Let assumptions in Theorem \6.1\ hold. Lin and Wei '16] proved 
that \6.1\) has no least energy solutions. Later, Sirakov \26jj proved that \6.1\) 
has a radially symmetric positive solution which has the least energy among all 
nontrivial radially symmetric solutions. Combining these with the introduction 
in Section 1, our results here are completely new. 

Define H r := H}(R N ) x H}(R N ) as a subspace of H := H 1 ^) x H 1 (R N ) 
with norm \\(u 1} u 2 )\\ 2 H = \\ui\\ 2 Xl + ||«2||f 2 , where 

H*(R N ) := {u G H 1 (R N ) : u is radially symmetric} , 

IMIi : = / (|Vu| 2 + X l u 2 )dx. 

Since the embedding H^(R N ) <—> L 4 (R N ) is compact, by replacing Hq(Q),H 
with Hr(R N ), H r respectively in all definitions appeared in Sections 2-4 and 
using the same notations, it is easy to see that all arguments (with trivial modi- 
fications) in Sections 2-4 hold for system (|6.ip except those in Step 2 of proving 
Theorems 11.11 and 11.31 Hence we only need to reprove Step 2 in the proofs of 
Theorems 11.11 and 11.31 The following ideas and arguments are quite different 
from those in Step 2 of proving Theorems 11.11 and 1 1 .31 and also can be used in 
the bounded domain case. 



Proof of Theorem 16.11 Assume by contradiction that there exists no G N 
such that (16.11) has only no radially symmetric sign-changing solutions. Fix any 
fc 2 > 2, we define 

I := max{c fcl ' fc2 : 2 < h < n + 2} + 1. 
For any fci G [2,n + 2] and < 6 < 2" 1 / 4 , similarly as (f2T23]l - (f2T24|) we define 
C £W? : = ™f . SU P Jsiu), (6.2) 

where 

(6.3) 



rfl M) := jAer (fei < fe2 > : supj^ < z j 



Lemma [2J] yields that T { ^ M) ^ 0, c^'J" is well defined and c^'J < c klM for 
each ki G [2, n + 2]. Noting that r^ I+1,fc2) C Tfj M \ we have 

2,fc 2 < 3,k 2 < < n + l.k 2 < n +2,k 2 10 a\ 

By repeating the arguments in Section 2 we can prove that, there exists 
Si G (0, 2" 1 / 4 ) such that for any 5 G (0, Si), 

r](t, u) € Vs whenever u G Mp n Vs, Jp(u) < I and t > 0, (6.5) 

and so c^ 1 /^ 2 is a radially symmetric sign-changing critical value of Ep for each 
fci G [2, no + 2] (that is, Ep has a radially symmetric sign-changing critical point 
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u with Ep{u) = Cpfg). Fix any a 5 <E (0, Si). By (|6.4p and our assumption that 
(|6.1[) has only no radially symmetric sign-changing solutions, there exists some 
2 < N x < n Q + 1 such that 

c ^,;,<5 — c p.i,s —■ c - 

Define 

JC := {u £ M : u sign-changing, Jp(u) = c, V(u) — 0}. (6.7) 

Then K. is finite. By (|2.17l) one has that <Ji(u) G K if tt G /C, that is, JC <Z F. 
Hence there exist k < no and {u m :l<m</c}c/C such that 

/C = {u m , ai(zt m ), cr 2 ("m), -w m : l<m<k}. 

Then there exist open neighborhoods Os m of u m in H r , such that any two of 
0$ mi , <?i(0 Sm2 ), a 2 (On m3 ) and -Oa m4 , where 1 < mi,m 2j ra 3 ,m4 < fc, are 
disjointed and 

k 

JCcO:= \J Sm U <7i (O a J U CT2 (Oa m ) U -0 Sm . 
Define a continuous map / : — > R \ {0} by 

if «eULi^i(Os m )U-O am . 

Then f(ai(u)) = —f[u) and f(a 2 (u)) = /(«)• By Tietze's extension theorem, 
there exists / € C(H r ,M) such that /|o = /• Define 

_ /(g)+/(^2(g))-/((Tl(g))-/(-fl) 

f w •- 4 

then F| = /, F(a 1 (u)) = -F(u) and F(a 2 (u)) = F{u). Define 



/C T := ^ u € .M : inf \\u — v\\ H < t 

I v£K 

Then we can take small r > such that /C2T C O. Recalling V(u) = in K, and 
/C finite, there exists C > such that 

||V(tZ)||ff < C, V«efe. (6.8) 

For any u € /C 2t , we have F(u) = f(u) ^ 0. That is F(JC 2t ) C K \ {0}. By 
(|6.7p and Lemma T2.9I there exists small e <E (0, 1) such that 

\\V(u)\\ 2 H >£, VueM/3\(lC T U Vs) satisfying \Jp(u) - c\ < 2e. (6.9) 
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Recall C in (gH5j) , we let 



a: =I m in{l,^!>. «>.1<l) 



By (JOJ-dOl) and (EH) we take A e r^ 1+1 ' fc2) such that 

sup < c^;"^ 1 ^ 2 +a£ = c + cue. (6-11) 

A\p s 

Let S := A \ IC 2t , then it is easy to check that B £ J 7 . We claim that j(B) > 
(Ni,k 2 ). If not, there exists g £ F^ lt i,^(B) such that g(u) ^ for any u £ B. 
By Tietze's extension theorem, there exists g = (g~i, g 2 ) £ C(H r ,R Nl ~ 1 x R fe2_1 ) 
such that g\ B = g. Define g = (51,52) G C^, M^ 1 " 1 x R fe2_1 ) by 

„ r-f\ ._ 9i(u) + gi(<r 2 (u j) -51(01(1?)) -.gi(-u) 



4 

52C") +52(0-1(1?)) -52(02(1?)) -g%{-u) 



then 5| B = 5; 5i(oi(i?)) = -g%{u) and 5i(<Xj(u)) = g l {u) for j ^ i. Finally we 
define G = {G X ,G 2 ) £ C(A, R^ 1 " 1 x R**-i) by 

Gi(fi) := (F(i2), 51(1?)) G R"^ 1-1-1-1 , G 2 (i?) := 52(1?) e R* 8 " 1 . 

By our constructions of F and 5, we have G € F(N 1 +i,k 2 ) C^)- Since 7(^4) > 
(iVi + l,fc 2 ), so G(t?) = for some w e A. If tt e fC 2T , then F(w) 7^ 0, 
a contradiction. So u e A \ JC 2t = B, and then g(u) = g(u) ^ 0, also a 
contradiction. Hence 7(B) > (Ni,k 2 ). Note that sup B Jp < sup A < I, we 
see that B £ Mp and B £ rffi M) . Then we can consider D := T](t/{2C),B), 
where is in Lemma [2.111 and C is in (|6.8[) . By Lemma [2.5I (m) and Lemma 
we have D € T^ 1 '^ and sup B < sup B Jp < I, that is D £ Tfl M) . 
Then we see from and (EH) that 

SUP Jy3 > C/3 , Is = C. 
D\Vs 

By Lemma ¥XM we can take u £ B such that t](t/(2C), u) £ D\Vs and 
c — ae < sup Jp — ae < Jp (17 (r/ (2(7), it)). 

Since Jp(r)(t,u)) < Jp(u) < I for any i > 0, (|6.5p yields r](t,u) # Vs for any 
t G [0,r/(2C)]. In particular, u^Vs and so 1 1|) yields Jp(u) < c + ae. Then 
for any t £ [0, r/ (2(7)], we have 

c- ae < Jp{rj{T/(2C),u)) < Jp(r)(t,u)) < Jp(u) < c + ae. 
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Recall that u £ B = A\K. 2r . If there exists T £ (0, r/(2C)) such that r?(T, u) € 
JC T , then there exist < t\ < t 2 < T such that r](ti,u) £ dK-2 T , rj(t 2 ,u) £ dJC T 
and rj(t,u) £ K,2 T \ K. T for any t £ {t\,t%). So we see from (|6.8p that 



t < \\r)(ti,u) - r)(t 2 ,u)\\H 



1 2 



V(rj(t,u))dt 



ti 



<2C(t 2 -t 1 ), 

H 



that is, r/(2C) < t 2 — t\ < T, a contradiction. Hence rj(t,u) £" JC T for any 
t £ (0,r/(2C)). Then as Step 1 in the proof of Theorem 1 we deduce from 
(EU) and (|CTJ)) that 

,r/(2C) 

c— as < J/3(t](t/(2C),u)) < Jp{u) — I CqS dt < c + as — 2ae = c — as, 

Jo 

a contradiction. Hence (|6.ip has infinite many radially symmetric sign-changing 
solutions. This completes the proof. □ 



Proof of Theorem 16.21 It suffices to prove that (|6.1[) has infinitely many 
semi-nodal solutions. This argument is similar as above, we omit the details. □ 
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